Output control using gesture input

ABSTRACT

An apparatus, including: at least one processor; and at least one memory storing computer program instructions, the at least one processor being configured to execute the computer program instructions to cause the apparatus at least to perform: detecting user input; selecting, in dependence upon the detected user input, at least a portion of an output of a user output device; determining, while the at least a portion of the output of the user output device is selected, whether a user makes a gesture input by moving the apparatus; and altering, in response to determining that the gesture input has been made by the user, the selected at least a portion of the output of the user output device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.13/518,189, filed Jun. 21, 2012, which is a national phase entry ofInternational Application No. PCT/CN2009/075819, filed Dec. 22, 2009,the entire contents of which are incorporated herein by reference.

FIELD

Embodiments of the present invention relate to controlling output usinggesture input. In particular, they relate to controlling the aural orvisual output of a portable electronic apparatus using gesture input.

BACKGROUND

The functionality and processing power provided by portable electronicdevices such as mobile telephones has increased significantly in recentyears. Some portable electronics devices now perform a diverse range offunctions. For example, some mobile telephones now provide camera andmusic playing functionality.

The introduction of this functionality means has led to some portableelectronic devices having a more complex user interface, making themdifficult to use.

BRIEF DESCRIPTION OF VARIOUS EMBODIMENTS

According to various, but not necessarily all, embodiments of theinvention there is provided an apparatus, comprising: at least oneprocessor; and at least one memory storing computer programinstructions, the at least one processor being configured to execute thecomputer program instructions to cause the apparatus at least toperform: detecting user input; selecting, in dependence upon thedetected user input, at least a portion of an output of a user outputdevice; determining, while the at least a portion of the output of theuser output device is selected, whether a user makes a gesture input bymoving the apparatus; and altering, in response to determining that thegesture input has been made by the user, the selected at least a portionof the output of the user output device.

According to various, but not necessarily all, embodiments of theinvention there is provided a method, comprising: detecting user input;selecting, in dependence upon the detected user input, at least aportion of an output of a user output device; determining, while the atleast a portion of the output of the user output device is selected,whether a user makes a gesture input by moving the apparatus; andaltering, in response to determining that the gesture input has beenmade by the user, the selected at least a portion of the output of theuser output device.

According to various, but not necessarily all, embodiments of theinvention there is provided a computer program comprising computerprogram instructions that, when executed by at least one processor,cause an apparatus at least to perform: detecting user input; selecting,in dependence upon the detected user input, at least a portion of anoutput of a user output device; determining, while the at least aportion of the output of the user output device is selected, whether auser makes a gesture input by moving the apparatus; and altering, inresponse to determining that the gesture input has been made by theuser, the selected at least a portion of the output of the user outputdevice.

According to various, but not necessarily all, embodiments of theinvention there is provided an apparatus, comprising: means fordetecting user input; means for selecting, in dependence upon thedetected user input, at least a portion of an output of a user outputdevice; means for determining, while the at least a portion of theoutput of the user output device is selected, whether a user makes agesture input by moving the apparatus; and means for altering, inresponse to determining that the gesture input has been made by theuser, the selected at least a portion of the output of the user outputdevice.

According to various, but not necessarily all, embodiments of theinvention there is provided an apparatus, comprising: a touch sensitivedisplay; at least one processor; and at least one memory storingcomputer program instructions, the at least one processor beingconfigured to execute the computer program instructions to cause theapparatus at least to perform: detecting user input at the touchsensitive display; selecting, in dependence upon the user input detectedat the touch sensitive display, content displayed on the touch sensitivedisplay; determining, while the content displayed on the touch sensitivedisplay is selected, whether a user makes a gesture input by moving theapparatus; and applying, in response to detecting the gesture input, avisual effect to the selected content displayed on the touch sensitivedisplay.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of various examples of embodiments of thepresent invention reference will now be made by way of example only tothe accompanying drawings in which:

FIG. 1 illustrates an apparatus;

FIG. 2 illustrates a further apparatus;

FIG. 3 illustrates a flow chart of a method;

FIG. 4 illustrates a first gesture input;

FIG. 5A illustrates an apparatus displaying an image;

FIG. 5B illustrates an apparatus displaying an image after a visualeffect has been applied;

FIG. 6 illustrates a second gesture input; and

FIG. 7 illustrates a plurality of visual effects.

DETAILED DESCRIPTION OF VARIOUS EMBODIMENTS

Embodiments of the invention relate to controlling the aural or visualoutput of an apparatus using gesture input. Gesture input, in the formof movement of the apparatus, causes the portion of aural or visualoutput to be altered. The aural or visual output may, for example, bealtered by applying an aural or visual effect.

The Figures illustrate an apparatus 10/30, comprising: at least oneprocessor 12; and at least one memory 14 storing computer programinstructions 18, the at least one processor 12 being configured toexecute the computer program instructions 18 to cause the apparatus10/30 at least to perform: detecting user input; selecting, independence upon the detected user input, at least a portion of an outputof a user output device 22/24/27; determining, while the at least aportion of the output of the user output device 22/24/27 is selected,whether a user makes a gesture input by moving the apparatus 10/30; andaltering, in response to determining that the gesture input has beenmade by the user, the selected at least a portion of the output of theuser output device 22/24/27.

FIG. 1 illustrates an apparatus 10. The apparatus may, for example, be achip or a chip-set. The apparatus 10 illustrated in FIG. 1 comprises aprocessor 12 and a memory 14. In alternative embodiments of theinvention, the apparatus 10 may comprise multiple processors.

The processor 12 is configured to read from and write to the memory 14.The processor 12 may also comprise an output interface via which dataand/or commands are output by the processor 12 and an input interfacevia which data and/or commands are input to the processor 12.

Although the memory 14 is illustrated as a single component it may beimplemented as one or more separate components, some or all of which maybe integrated/removable and/or may providepermanent/semi-permanent/dynamic/cached storage.

The memory 14 stores a computer program 16 comprising computer programinstructions 18 that control the operation of the apparatus 10/30 whenloaded into the processor 12. The computer program instructions 18provide the logic and routines that enables the apparatus 10/30 toperform the method illustrated in FIG. 3. The processor 12 by readingthe memory 14 is able to load and execute the computer programinstructions 18.

The computer program 16 may arrive at the apparatus 10 via any suitabledelivery mechanism 40. The delivery mechanism 40 may be, for example, atangible computer-readable storage medium, a computer program product, amemory device, a record medium such as a CD-ROM, DVD or Blu-Ray disc, orany article of manufacture that tangibly embodies the computer program16. The delivery mechanism 40 may be a signal configured to reliablytransfer the computer program 16.

FIG. 2 illustrates a further apparatus 30. The apparatus 30 illustratedin FIG. 2 may, for example, be a portable electronic apparatus such as amobile telephone, a personal music player, a personal digital assistant,or a camera.

The apparatus 30 illustrated in FIG. 2 comprises the apparatus 10illustrated in FIG. 1. The apparatus 30 further comprises a housing 28which houses: the processor 12, the memory 14, a user input device 20,first and second user output devices 22, 27, and at least oneaccelerometer 26. The elements 12, 14, 20, 22, 26 and 27 are co-locatedwithin the housing 28.

The processor 12 is configured to provide an output to the first andsecond user output devices 22, 27 and to receive an input from the userinput device 20 and the one or more accelerometers 26. The elements 12,14, 20, 22, 26 and 27 are operationally coupled and any number orcombination of intervening elements can exist (including no interveningelements).

The first user output device 22 may, for example, be a display. In someembodiments of the invention, the user input device 20 and the firstuser output device 22 are combined in a touch sensitive display 24. Inother embodiments, the first user output device 22 and the user inputdevice 20 are separate. For example, the user input device 20 may be (orcomprise) a keypad and/or a microphone. The second user output device 27may, for example, be a loudspeaker.

An exemplary method according to embodiments of the invention will nowbe described in relation to FIGS. 3, 4, 5A and 5B. In this example,gesture input is used to apply a visual effect.

FIG. 5A illustrates the apparatus 30 displaying an image 54 on a touchsensitive display 24. The image 54 includes first and second content 55,57. In this example, the first content is a person 55 in the foregroundof the image and the second content is the background of the image 57.

At block 150 of FIG. 3, the processor 12 detects user input at the userinput device 20. Since, in this particular example, the user inputdevice 20 and the first user output device 22 are combined in a touchsensitive display 24, the user provides the input by touching the touchsensitive display 24.

In this particular example, the user provides an input indicating thecontent on the display 24 that he wishes to remain unaltered. The usertouches the first content 55, which in this example is the person in theforeground of the image 54.

At block 160 of FIG. 3, the processor 12 selects a portion of the outputof the display 24 for alteration. The selection made by the processor 12depends upon the user input provided by the user. In this example, sincethe user has provided input indicating that he wishes the first content(the person) 55 to remain unaltered, the processor 12 selects the secondcontent (the background) 57 as the portion for alteration.

In some embodiments of the invention, the processor 12 may distinguishthe second content 57 from the first content 55 by merely selecting allof the parts of the image that the user does not touch. In otherembodiments of the invention, the processor 12 may perform the selectionby performing a processing algorithm to distinguish the second content57 from the first content 55. For example, the processing algorithm maycomprise identifying differences in contrast in the area of the image 54identified by the input from the user, in order to distinguish thesecond content 57 from the first content 55.

The processor 12 may or may not control the display to display anindication that the second content 57 has been selected for alteration(for example, by highlighting).

At block 170 of FIG. 3, the processor 12 determines, while the portion57 of the output 54 of the display 24 is selected, whether the usermakes a gesture input by moving the apparatus 10/30. One example of agesture input is illustrated in FIG. 4.

The gesture input illustrated in FIG. 4 is a shaking gesture whichinvolves placing the apparatus 10/30 into reciprocal motion. The userinitially moves the apparatus 10/30 in a first direction 60, from afirst position 75 to a second position 65. The user then moves theapparatus 10/30 in a second direction 70, from the second position 65 toapproximately the first position 75. The second direction 70 issubstantially opposite to the first direction 60. The user continuallymoves the apparatus 10/30 from the first position 75 to the secondposition 65, and vice versa, such that the apparatus is in reciprocalmotion. While the gesture input of FIG. 4 is being performed by theuser, the apparatus 10/30 moves to a greater extent horizontally thanvertically.

The processor 12 determines that the gesture input illustrated in FIG. 4has been made by processing inputs received from the accelerometer(s)26. For example, the processor 12 may determine that the gesture inputhas been made if the acceleration measured by the accelerometer(s) 26 isconsistent with the gesture input illustrated in FIG. 4.

For example, the processor 12 may determine that the accelerationmeasured by the accelerometers is consistent with the gesture inputillustrated in FIG. 4 by analyzing one or more acceleration vectorsmeasured by the accelerometer(s) 26 over a period of time. The processor12 may also determine whether the acceleration of the apparatus 10/30exceeds a threshold value.

At block 180 of FIG. 3, in response to determining that a gesture inputhas been made by a user, the processor 12 alters the selected portion 57of the output 54 of the display 24. In this example, the processor 12alters the selected second content 57 by applying a visual effect to it.FIG. 5B illustrates the apparatus 30 after a visual effect has beenapplied to the selected second content 57. In this example, the visualeffect is a “blurring effect”, in which the second content 57 isblurred, such that it appears as if the first content 55 is in motion.

The alteration of the selected portion 57 of the output 54 of thedisplay 54 may depend upon the magnitude of the acceleration measured bythe accelerometer(s) 26. For example, the “blurring effect” may be morepronounced if the apparatus 10/30 is shaken more vigorously.

Advantageously, embodiments of the invention provide a method ofenabling a user to alter the output of a user output device, such as adisplay, in a straightforward and intuitive manner.

The processor 12 may be configured to alter the output 54 of the display24 differently, depending upon the gesture input provided by the user.For instance, the processor 12 may be configured to alter the output 54by performing a first processing algorithm on the output 54 in responseto determining that a first gesture input has been made, or byperforming a second processing algorithm on the output 54 in response todetermining that a second gesture input has been made.

As explained above, while the gesture input of FIG. 4 is being performedby the user, the apparatus 30 moves to a greater extent horizontallythan vertically. Consequently, the processor 12 applies a “blurringeffect” to the second content 57 in the FIG. 5B example that makes thefirst content 55 appear as if it is in horizontal motion.

FIG. 6 illustrates a shaking gesture input that is different to thatillustrated in FIG. 4. In the FIG. 6 example, the user moves theapparatus 30 in a first direction 80, from a first position 95 to asecond position 85. The user then moves the apparatus 30 in a seconddirection 90, from the second position 85 to approximately the firstposition 95. The second direction 90 is substantially opposite to thefirst direction 80. The user continually moves the apparatus 10/30 fromthe first position 95 to the second position 85, and vice versa, suchthat the apparatus 30 is in reciprocal motion. While the gesture inputof FIG. 6 is being performed by the user, the apparatus 30 moves to agreater extent vertically than horizontally.

The processor 12 may determine whether the first gesture input or thesecond gesture input has been made by analyzing one or more accelerationvectors measured by the accelerometer(s) 26 over a period of time(during a gesture input). For example, the processor 12 may analyze theone or more measured acceleration vectors to determine whether movementof the apparatus 30 during the gesture input is greater horizontally orvertically. The processor 12 may determine that the first gesture input(illustrated in FIG. 4) has been made by the user if the apparatus 30moves to a greater extent horizontally than vertically. The processor 12may determine that the second gesture input (illustrated in FIG. 6) hasbeen made by the user if the apparatus 30 moves to a greater extentvertically than horizontally.

In some embodiments of the invention, if the processor 12 determinesthat the gesture input of FIG. 6 has been made (rather than the gestureinput of FIG. 4), the processor 12 applies a different visual effect tothe second content 57 of FIG. 5A compared to that illustrated in FIG.5B. For example, the processor may apply a “blurring effect” makes thefirst content 55 appear as if it is in vertical motion.

The temporal relationship between: i) the user input at block 150 thatdetermines which portion of the output 54 of the display 24 is selectedby the processor 12 for alteration at block 160, and ii) the gestureinput detected at block 170 may be different, depending upon theimplementation of the invention.

In a first implementation, the user provides the input detected at block150 by touching the display 24 with a digit, and then releases the digitfrom the display 24 before providing the gesture input of FIG. 4 or FIG.6, which is detected at block 170.

In second implementation, the user provides the input detected at block150 by touching the display 24 with a digit. The user continues to holdthe digit in place 24 prior to providing the gesture input of FIG. 4 orFIG. 6 and while providing the gesture input of FIG. 4 or FIG. 6, whichis detected at block 170.

In a third implementation, the user begins providing the gesture inputof FIG. 4 or FIG. 6, and then provides the touch input detected at block150 by touching the display 24 while providing the gesture input. It maybe that the gesture input is detected at block 170 in response todetection of the touch input at block 150.

Various modifications to the method described above are possible. Forexample, in alternative embodiments of the invention the user may, forexample, provide input indicating the content on the display 24 that hewishes to alter (rather than that which he wishes to remain unaltered).In these embodiments of the invention, the user touches the secondcontent (the background) 57 illustrated in FIG. 5A, in order to apply avisual effect to it. Also, the user input detected at block 150 may beprovided at a user input device other than a touch sensitive display.For example the user input device may be a keypad, separate to thedisplay of the apparatus 30.

In some embodiments of the invention, the visual effect may be appliedto the whole of the image 54 rather than merely a portion 57 of theimage 54.

In some exemplary embodiments, the gesture input provided by the userthat causes the processor 12 to alter at least a portion of the outputof the display 24 need not be a shaking gesture. For example, in someembodiments of the invention, the processor 12 may detect a gestureinput which merely involves moving the apparatus 30 in a singledirection (such as the direction 60 illustrated in FIG. 4 or,alternatively, the direction 90 illustrated in FIG. 6). The processor 12may determine whether this gesture input has been provided bydetermining whether the acceleration measured by the accelerometer(s) 26exceeds a threshold value.

FIG. 7 illustrates four examples 100, 200, 300 and 400 of how the outputof a display may be altered by the processor 12 in response todetermining that a gesture input has been made at block 170.

In the first example 100 in FIG. 7, the user provides user input (forinstance, touch input) indicating that first content 102 is to remainunaltered. In response to detecting the user input at block 150, theprocessor 12 selects second content 104, 106, 108 and 110 foralteration. In response to determining that a user has made a gestureinput by moving the apparatus 30 at block 170, the processor 12 appliesa visual effect by changing the relative positions of portions of thesecond content 104, 106, 108 and 110 on the display 22/24.

The second example 200 in FIG. 7 relates to an image depicting anegative comprising three sub-images. A user provides input (forinstance, touch input) at block 150 indicating that a sub-image 203 isto be altered. In response to determining that a user has made a gestureinput by moving the apparatus 30 at block 170, the processor 12 appliesa visual effect by changing the background 205 of the sub-image 203.

The third example 300 in FIG. 7 illustrates a web-page 301 beingdisplayed on the display 22/24 of the apparatus 30. The text of the webpage 301 is legible on the display 22/24. A user provides input (forinstance, touch input) indicating that a portion 302 of the web-page 301is to be altered. At block 150, the processor 12 selects the relevantportion 302 of the web-page in response to detecting the user input. Inresponse to determining that a user has made a gesture input by movingthe apparatus 30 at block 170, the processor 12 applies a visual effectby magnifying the selected portion 302 of the web-page 301.

The fourth example 400 in FIG. 7 relates to text displayed on thedisplay 22/24 of the apparatus 30. The text may, for example, be part ofa text message. A user provides input (for instance, touch input)indicating a portion 402 of text to be altered. At block 150, theprocessor 12 selects the relevant portion 402 of the text in response todetecting the user input. In response to determining that a user hasmade a gesture input by moving the apparatus 30 at block 170, theprocessor 12 applies a visual effect by enlarging the selected portion402 of text.

The blocks illustrated in FIG. 3 may represent steps in a method and/orsections of code in the computer program 16. The illustration of aparticular order to the blocks does not necessarily imply that there isa required or preferred order for the blocks and the order andarrangement of the block may be varied. Furthermore, it may be possiblefor some steps to be omitted.

Although embodiments of the present invention have been described in thepreceding paragraphs with reference to various examples, it should beappreciated that modifications to the examples given can be made withoutdeparting from the scope of the invention as claimed. For example, inthe examples illustrated in FIGS. 5A, 5B and FIG. 7, the processor 12alters at least a portion of an image displayed on a display 22/24 byapplying a visual effect. In these embodiments of the invention, thedisplay is the “user output device” that provides the output foralteration.

In some other embodiments of the invention, the user output device thatprovides the output for alteration is not a display. Instead, the useroutput device providing the output for alteration may be the loudspeaker27. For example, at block 150 of FIG. 3, a user may provide user inputthat indicates that at least a portion of the aural output of theloudspeaker is to be altered. For example, the volume of the loudspeaker27 could be altered, or merely the bass or treble output of theloudspeaker 27.

In response to determining that the user has made an appropriate gestureinput by moving the apparatus 30 at block 170, the processor 12 altersat least a portion of the aural output of the loudspeaker 27 (forexample, by altering the volume or the bass or treble output).

References to ‘a tangible computer-readable storage medium’, ‘a computerprogram product’, a ‘computer’, and a ‘processor’ etc. should beunderstood to encompass not only computers having differentarchitectures such as single/multi-processor architectures andsequential (Von Neumann)/parallel architectures but also specializedcircuits such as field-programmable gate arrays (FPGA), applicationspecific circuits (ASIC), signal processing devices and other devices.References to computer program, instructions, code etc. should beunderstood to encompass software for a programmable processor orfirmware such as, for example, the programmable content of a hardwaredevice whether instructions for a processor, or configuration settingsfor a fixed-function device, gate array or programmable logic deviceetc.

Features described in the preceding description may be used incombinations other than the combinations explicitly described.

Although functions have been described with reference to certainfeatures, those functions may be performable by other features whetherdescribed or not.

Although features have been described with reference to certainembodiments, those features may also be present in other embodimentswhether described or not.

Whilst endeavoring in the foregoing specification to draw attention tothose features of the invention believed to be of particular importanceit should be understood that the Applicant claims protection in respectof any patentable feature or combination of features hereinbeforereferred to and/or shown in the drawings whether or not particularemphasis has been placed thereon.

1. An apparatus, comprising: at least one processor; and at least onememory storing computer program instructions configured, working withthe at least one processor, to cause the apparatus at least to perform:detecting user input indicating that a characteristic of aural output ofa loudspeaker is to be altered; determining whether a user makes agesture input by moving the apparatus; and in response to determiningthat the gesture input has been made by the user, altering thecharacteristic of the aural output of the loudspeaker as indicated bythe user input.
 2. An apparatus according to claim 1, wherein thecharacteristic of aural output to be altered comprises a volume of theaural output of the loudspeaker.
 3. An apparatus according to claim 1,wherein the characteristic of aural output to be altered comprises abass or a treble of the aural output of the loudspeaker.
 4. An apparatusaccording to claim 1, wherein the characteristic of the aural output isaltered by performing a first processing algorithm in response todetermining that a first gesture input has been made by the user, andaltered by performing a second processing algorithm in response todetermining that a second gesture input has been made by the user.
 5. Anapparatus according to claim 1, wherein the apparatus is caused todetermine whether a gesture input is made by the user by detectingwhether acceleration of the apparatus exceeds a threshold accelerationvalue.
 6. An apparatus according to claim 1, wherein the apparatus iscaused to determine whether a user makes a gesture input by detecting aninput from at least one accelerometer co-located with the at least oneprocessor and the at least one memory.
 7. An apparatus according toclaim 1, wherein the gesture input comprises movement of the apparatusin a first direction and a second direction, different to the firstdirection.
 8. An apparatus according to claim 7, wherein the apparatusis caused to determine whether a user makes a gesture input bydetermining whether the apparatus is in reciprocating motion.
 9. Amethod, comprising: detecting user input indicating that acharacteristic of aural output of a loudspeaker is to be altered;determining whether a user makes a gesture input by moving an apparatus;and in response to determining that the gesture input has been made bythe user, altering the characteristic of the aural output of theloudspeaker as indicated by the user input.
 10. A method according toclaim 9, wherein the characteristic of aural output to be alteredcomprises a volume of the aural output of the loudspeaker.
 11. A methodaccording to claim 9, wherein the characteristic of aural output to bealtered comprises a bass or a treble of the aural output of theloudspeaker.
 12. A method according to claim 9, wherein thecharacteristic of the aural output is altered by performing a firstprocessing algorithm in response to determining that a first gestureinput has been made by the user, and altered by performing a secondprocessing algorithm in response to determining that a second gestureinput has been made by the user.
 13. A method according to claim 9,wherein determining whether a gesture input is made by the usercomprises detecting whether acceleration of the apparatus exceeds athreshold acceleration value.
 14. A method according to claim 9, whereindetermining whether a user makes a gesture input comprises detecting aninput from at least one accelerometer associated with the apparatus. 15.A method according to claim 9, wherein the gesture input comprisesmovement of the apparatus in a first direction and a second direction,different to the first direction.
 16. A method according to claim 15,wherein determining whether a user makes a gesture input comprisesdetermining whether the apparatus is in reciprocating motion.
 17. Anon-transitory computer readable medium storing a computer programcomprising computer program instructions configured, when working withat least one processor, to cause an apparatus at least to perform:detecting user input indicating that a characteristic of aural output ofa loudspeaker is to be altered; determining whether a user makes agesture input by moving the apparatus; and in response to determiningthat the gesture input has been made by the user, altering thecharacteristic of the aural output of the loudspeaker as indicated bythe user input.
 18. A computer readable medium according to claim 17,wherein the characteristic of aural output to be altered comprises avolume of the aural output of the loudspeaker.
 19. A computer readablemedium according to claim 17, wherein the characteristic of aural outputto be altered comprises a bass or a treble of the aural output of theloudspeaker.
 20. A computer readable medium according to claim 17,wherein the characteristic of the aural output is altered by performinga first processing algorithm in response to determining that a firstgesture input has been made by the user, and altered by performing asecond processing algorithm in response to determining that a secondgesture input has been made by the user.